Have you ever wondered what happens when a C program is compiled and executed on a system? This class will investigate the life of a binary from birth as C source code to death as a process running in memory being terminated.
Topics will include but are not limited to: • Scanning and tokenizing source code. • Parsing a grammar and outputting assembly code. • Different targets for x86 assembly object files generation. (E.g. relocatable vs. position independent code). • Linking object files together to create a well-formed binary. • Detailed description of the Windows PE binary formats. • How Windows loads a binary into memory and links it on the fly before executing it.
Along the way we will discuss the relevance of security at different stages of a binary’s life, from how viruses *really* work, to the way which malware “packers” duplicate OS process execution functionality, to the benefit of a security-enhanced OS loader which implements address space layout randomization (ASLR).
Lab work will include: • Manipulating compiler options to change the type of assembly which is output • Manipulating linker options to change the structure of binary formats • Reading and understanding PE files with PEView • Using WinDbg to watch the loader resolve imports in an executable • Using Thread Local Storage (TLS) to obfuscate control flow and serve as a basic anti-debug mechanism • Creating a simple example virus for PE • Analyze the changes made to the binary format when a file is packed with UPX • Using the rootkit technique of Import Address Table (IAT) hooking to subvert the integrity of a program’s calls to external libraries, allowing processes to be hidden.
The prerequisites for this class are a basic understanding of C programming and compilation. This class will be recommended for a later class on rootkits, and required for a later class on malware analysis.